

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Tracing Ceph With LTTng &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/doctools.js"></script>
        <script src="../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="BlueStore Internals" href="../bluestore/" />
    <link rel="prev" title="Ceph 如何均衡（读写、容量）" href="../balancer-design/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../internals/">Ceph 内幕</a></li>
      <li class="breadcrumb-item active">Tracing Ceph With LTTng</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../_sources/dev/blkin.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developer_guide/">开发者指南</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../internals/">Ceph 内幕</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../balancer-design/">Ceph 如何均衡（读写、容量）</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Tracing Ceph With LTTng</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#configuring-ceph-with-lttng">Configuring Ceph with LTTng</a></li>
<li class="toctree-l3"><a class="reference internal" href="#testing-trace">Testing Trace</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#tracing-ceph-with-blkin">Tracing Ceph With Blkin</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#configuring-ceph-with-blkin">Configuring Ceph with Blkin</a></li>
<li class="toctree-l3"><a class="reference internal" href="#testing-blkin">Testing Blkin</a></li>
<li class="toctree-l3"><a class="reference internal" href="#install-zipkin">Install  Zipkin</a></li>
<li class="toctree-l3"><a class="reference internal" href="#show-ceph-s-blkin-traces-in-zipkin-web">Show Ceph’s Blkin Traces in Zipkin-web</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../bluestore/">BlueStore Internals</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph_krb_auth/">如何配置好 Ceph Kerberos 认证的详细文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-mirroring/">CephFS Mirroring</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-reclaim/">CephFS Reclaim Interface</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-snapshots/">CephFS 快照</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx/">Cephx</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx_protocol/">Cephx 认证协议详细阐述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config/">配置管理系统</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config-key/">config-key layout</a></li>
<li class="toctree-l2"><a class="reference internal" href="../context/">CephContext</a></li>
<li class="toctree-l2"><a class="reference internal" href="../continuous-integration/">Continuous Integration Architecture</a></li>
<li class="toctree-l2"><a class="reference internal" href="../corpus/">资料库结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cpu-profiler/">Oprofile 的安装</a></li>
<li class="toctree-l2"><a class="reference internal" href="../crush-msr/">CRUSH MSR (Multi-step Retry)</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cxx/">C++17 and libstdc++ ABI</a></li>
<li class="toctree-l2"><a class="reference internal" href="../deduplication/">去重</a></li>
<li class="toctree-l2"><a class="reference internal" href="../delayed-delete/">CephFS delayed deletion</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployment/">开发集群的部署</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployment/#id5">在同一机器上部署多套开发集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../development-workflow/">开发流程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../documenting/">为 Ceph 写作文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dpdk/">Ceph messenger DPDKStack</a></li>
<li class="toctree-l2"><a class="reference internal" href="../encoding/">序列化（编码、解码）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../erasure-coded-pool/">纠删码存储池</a></li>
<li class="toctree-l2"><a class="reference internal" href="../file-striping/">File striping</a></li>
<li class="toctree-l2"><a class="reference internal" href="../freebsd/">FreeBSD Implementation details</a></li>
<li class="toctree-l2"><a class="reference internal" href="../generatedocs/">Ceph 文档的构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../health-reports/">Health Reports</a></li>
<li class="toctree-l2"><a class="reference internal" href="../iana/">IANA 号</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/">Testing changes to the Linux Kernel CephFS driver</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/#step-one-build-the-kernel">Step One: build the kernel</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/#step-two-create-a-vm">Step Two: create a VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/#step-three-networking-the-vm">Step Three: Networking the VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kubernetes/">Hacking on Ceph in Kubernetes with Rook</a></li>
<li class="toctree-l2"><a class="reference internal" href="../libcephfs_proxy/">Design of the libcephfs proxy</a></li>
<li class="toctree-l2"><a class="reference internal" href="../libs/">库体系结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logging/">集群日志的用法</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logs/">调试日志</a></li>
<li class="toctree-l2"><a class="reference internal" href="../macos/">在 MacOS 上构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mempool_accounting/">What is a mempool?</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mempool_accounting/#some-common-mempools-that-we-can-track">Some common mempools that we can track</a></li>
<li class="toctree-l2"><a class="reference internal" href="../messenger/">Messenger notes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-bootstrap/">Monitor bootstrap</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-elections/">Monitor Elections</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-on-disk-formats/">ON-DISK FORMAT</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-osdmap-prune/">FULL OSDMAP VERSION PRUNING</a></li>
<li class="toctree-l2"><a class="reference internal" href="../msgr2/">msgr2 协议（ msgr2.0 和 msgr2.1 ）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-encoding/">Network Encoding</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-protocol/">网络协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../object-store/">对象存储架构概述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd-class-path/">OSD class path issues</a></li>
<li class="toctree-l2"><a class="reference internal" href="../peering/">互联</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf/">Using perf</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_counters/">性能计数器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_histograms/">Perf histograms</a></li>
<li class="toctree-l2"><a class="reference internal" href="../placement-group/">PG （归置组）说明</a></li>
<li class="toctree-l2"><a class="reference internal" href="../quick_guide/">开发者指南（快速）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rados-client-protocol/">RADOS 客户端协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-diff/">RBD 增量备份</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-export/">RBD Export &amp; Import</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-layering/">RBD Layering</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-checklists/">Release checklists</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-process/">Ceph Release Process</a></li>
<li class="toctree-l2"><a class="reference internal" href="../seastore/">SeaStore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../sepia/">Sepia 社区测试实验室</a></li>
<li class="toctree-l2"><a class="reference internal" href="../session_authentication/">Session Authentication for the Cephx Protocol</a></li>
<li class="toctree-l2"><a class="reference internal" href="../testing/">测试笔记</a></li>
<li class="toctree-l2"><a class="reference internal" href="../versions/">Public OSD Version</a></li>
<li class="toctree-l2"><a class="reference internal" href="../vstart-ganesha/">NFS CephFS-RGW Developer Guide</a></li>
<li class="toctree-l2"><a class="reference internal" href="../wireshark/">Wireshark Dissector</a></li>
<li class="toctree-l2"><a class="reference internal" href="../zoned-storage/">Zoned Storage Support</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd_internals/">OSD 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mds_internals/">MDS 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../radosgw/">RADOS 网关开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph-volume/">ceph-volume 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../crimson/">Crimson developer documentation</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="tracing-ceph-with-lttng">
<h1>Tracing Ceph With LTTng<a class="headerlink" href="#tracing-ceph-with-lttng" title="Permalink to this heading"></a></h1>
<section id="configuring-ceph-with-lttng">
<h2>Configuring Ceph with LTTng<a class="headerlink" href="#configuring-ceph-with-lttng" title="Permalink to this heading"></a></h2>
<p>Use -DWITH_LTTNG option (default: ON):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">./</span><span class="n">do_cmake</span> <span class="o">-</span><span class="n">DWITH_LTTNG</span><span class="o">=</span><span class="n">ON</span>
</pre></div>
</div>
<p>Config option for tracing must be set to true in ceph.conf.
Following options are currently available:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">bluestore_tracing</span>
<span class="n">event_tracing</span> <span class="p">(</span><span class="o">-</span><span class="n">DWITH_EVENTTRACE</span><span class="p">)</span>
<span class="n">osd_function_tracing</span> <span class="p">(</span><span class="o">-</span><span class="n">DWITH_OSD_INSTRUMENT_FUNCTIONS</span><span class="p">)</span>
<span class="n">osd_objectstore_tracing</span> <span class="p">(</span><span class="n">actually</span> <span class="n">filestore</span> <span class="n">tracing</span><span class="p">)</span>
<span class="n">rbd_tracing</span>
<span class="n">osd_tracing</span>
<span class="n">rados_tracing</span>
<span class="n">rgw_op_tracing</span>
<span class="n">rgw_rados_tracing</span>
</pre></div>
</div>
</section>
<section id="testing-trace">
<h2>Testing Trace<a class="headerlink" href="#testing-trace" title="Permalink to this heading"></a></h2>
<p>Start LTTng daemon:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lttng</span><span class="o">-</span><span class="n">sessiond</span> <span class="o">--</span><span class="n">daemonize</span>
</pre></div>
</div>
<p>Run vstart cluster with enabling trace options:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">../</span><span class="n">src</span><span class="o">/</span><span class="n">vstart</span><span class="o">.</span><span class="n">sh</span> <span class="o">-</span><span class="n">d</span> <span class="o">-</span><span class="n">n</span> <span class="o">-</span><span class="n">l</span> <span class="o">-</span><span class="n">e</span> <span class="o">-</span><span class="n">o</span> <span class="s2">&quot;osd_tracing = true&quot;</span>
</pre></div>
</div>
<p>List available tracepoints:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lttng</span> <span class="nb">list</span> <span class="o">--</span><span class="n">userspace</span>
</pre></div>
</div>
<p>You will get something like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">UST</span> <span class="n">events</span><span class="p">:</span>
<span class="o">-------------</span>
<span class="n">PID</span><span class="p">:</span> <span class="mi">100859</span> <span class="o">-</span> <span class="n">Name</span><span class="p">:</span> <span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">ceph</span><span class="o">-</span><span class="n">osd</span>
    <span class="n">pg</span><span class="p">:</span><span class="n">queue_op</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_DEBUG_LINE</span> <span class="p">(</span><span class="mi">13</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
    <span class="n">osd</span><span class="p">:</span><span class="n">do_osd_op_post</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_DEBUG_LINE</span> <span class="p">(</span><span class="mi">13</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
    <span class="n">osd</span><span class="p">:</span><span class="n">do_osd_op_pre_unknown</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_DEBUG_LINE</span> <span class="p">(</span><span class="mi">13</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
    <span class="n">osd</span><span class="p">:</span><span class="n">do_osd_op_pre_copy_from</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_DEBUG_LINE</span> <span class="p">(</span><span class="mi">13</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
    <span class="n">osd</span><span class="p">:</span><span class="n">do_osd_op_pre_copy_get</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_DEBUG_LINE</span> <span class="p">(</span><span class="mi">13</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
    <span class="o">...</span>
</pre></div>
</div>
<p>Create tracing session, enable tracepoints and start trace:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lttng</span> <span class="n">create</span> <span class="n">trace</span><span class="o">-</span><span class="n">test</span>
<span class="n">lttng</span> <span class="n">enable</span><span class="o">-</span><span class="n">event</span> <span class="o">--</span><span class="n">userspace</span> <span class="n">osd</span><span class="p">:</span><span class="o">*</span>
<span class="n">lttng</span> <span class="n">start</span>
</pre></div>
</div>
<p>Perform some ceph operatin:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">rados</span> <span class="n">bench</span> <span class="o">-</span><span class="n">p</span> <span class="n">ec</span> <span class="mi">5</span> <span class="n">write</span>
</pre></div>
</div>
<p>Stop tracing and view result:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lttng</span> <span class="n">stop</span>
<span class="n">lttng</span> <span class="n">view</span>
</pre></div>
</div>
<p>Destroy tracing session:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lttng</span> <span class="n">destroy</span>
</pre></div>
</div>
</section>
</section>
<section id="tracing-ceph-with-blkin">
<h1>Tracing Ceph With Blkin<a class="headerlink" href="#tracing-ceph-with-blkin" title="Permalink to this heading"></a></h1>
<p>Ceph can use Blkin, a library created by Marios Kogias and others,
which enables tracking a specific request from the time it enters
the system at higher levels till it is finally served by RADOS.</p>
<p>In general, Blkin implements the <a class="reference external" href="http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf">Dapper</a> tracing semantics
in order to show the causal relationships between the different
processing phases that an IO request may trigger. The goal is an
end-to-end visualisation of the request’s route in the system,
accompanied by information concerning latencies in each processing
phase. Thanks to LTTng this can happen with a minimal overhead and
in realtime. The LTTng traces can then be visualized with Twitter’s
<a class="reference external" href="https://zipkin.io/">Zipkin</a>.</p>
<section id="configuring-ceph-with-blkin">
<h2>Configuring Ceph with Blkin<a class="headerlink" href="#configuring-ceph-with-blkin" title="Permalink to this heading"></a></h2>
<p>Use -DWITH_BLKIN option (which requires -DWITH_LTTNG):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">./</span><span class="n">do_cmake</span> <span class="o">-</span><span class="n">DWITH_LTTNG</span><span class="o">=</span><span class="n">ON</span> <span class="o">-</span><span class="n">DWITH_BLKIN</span><span class="o">=</span><span class="n">ON</span>
</pre></div>
</div>
<p>Config option for blkin must be set to true in ceph.conf.
Following options are currently available:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">rbd_blkin_trace_all</span>
<span class="n">osd_blkin_trace_all</span>
<span class="n">osdc_blkin_trace_all</span>
</pre></div>
</div>
</section>
<section id="testing-blkin">
<h2>Testing Blkin<a class="headerlink" href="#testing-blkin" title="Permalink to this heading"></a></h2>
<p>It’s easy to test Ceph’s Blkin tracing. Let’s assume you don’t have
Ceph already running, and you compiled Ceph with Blkin support but
you didn’t install it. Then launch Ceph with the <code class="docutils literal notranslate"><span class="pre">vstart.sh</span></code> script
in Ceph’s src directory so you can see the possible tracepoints.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">OSD</span><span class="o">=</span><span class="mi">3</span> <span class="n">MON</span><span class="o">=</span><span class="mi">3</span> <span class="n">RGW</span><span class="o">=</span><span class="mi">1</span> <span class="o">../</span><span class="n">src</span><span class="o">/</span><span class="n">vstart</span><span class="o">.</span><span class="n">sh</span> <span class="o">-</span><span class="n">n</span> <span class="o">-</span><span class="n">o</span> <span class="s2">&quot;rbd_blkin_trace_all&quot;</span>
<span class="n">lttng</span> <span class="nb">list</span> <span class="o">--</span><span class="n">userspace</span>
</pre></div>
</div>
<p>You’ll see something like the following::</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">UST</span> <span class="n">events</span><span class="p">:</span>
<span class="o">-------------</span>
<span class="n">PID</span><span class="p">:</span> <span class="mi">8987</span> <span class="o">-</span> <span class="n">Name</span><span class="p">:</span> <span class="o">./</span><span class="n">ceph</span><span class="o">-</span><span class="n">osd</span>
      <span class="n">zipkin</span><span class="p">:</span><span class="n">timestamp</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_WARNING</span> <span class="p">(</span><span class="mi">4</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
      <span class="n">zipkin</span><span class="p">:</span><span class="n">keyval_integer</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_WARNING</span> <span class="p">(</span><span class="mi">4</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
      <span class="n">zipkin</span><span class="p">:</span><span class="n">keyval_string</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_WARNING</span> <span class="p">(</span><span class="mi">4</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
      <span class="n">lttng_ust_tracelog</span><span class="p">:</span><span class="n">TRACE_DEBUG</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_DEBUG</span> <span class="p">(</span><span class="mi">14</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>

<span class="n">PID</span><span class="p">:</span> <span class="mi">8407</span> <span class="o">-</span> <span class="n">Name</span><span class="p">:</span> <span class="o">./</span><span class="n">ceph</span><span class="o">-</span><span class="n">mon</span>
      <span class="n">zipkin</span><span class="p">:</span><span class="n">timestamp</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_WARNING</span> <span class="p">(</span><span class="mi">4</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
      <span class="n">zipkin</span><span class="p">:</span><span class="n">keyval_integer</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_WARNING</span> <span class="p">(</span><span class="mi">4</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
      <span class="n">zipkin</span><span class="p">:</span><span class="n">keyval_string</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_WARNING</span> <span class="p">(</span><span class="mi">4</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>
      <span class="n">lttng_ust_tracelog</span><span class="p">:</span><span class="n">TRACE_DEBUG</span> <span class="p">(</span><span class="n">loglevel</span><span class="p">:</span> <span class="n">TRACE_DEBUG</span> <span class="p">(</span><span class="mi">14</span><span class="p">))</span> <span class="p">(</span><span class="nb">type</span><span class="p">:</span> <span class="n">tracepoint</span><span class="p">)</span>

<span class="o">...</span>
</pre></div>
</div>
<p>Next, stop Ceph so that the tracepoints can be enabled.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">../</span><span class="n">src</span><span class="o">/</span><span class="n">stop</span><span class="o">.</span><span class="n">sh</span>
</pre></div>
</div>
<p>Start up an LTTng session and enable the tracepoints.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lttng</span> <span class="n">create</span> <span class="n">blkin</span><span class="o">-</span><span class="n">test</span>
<span class="n">lttng</span> <span class="n">enable</span><span class="o">-</span><span class="n">event</span> <span class="o">--</span><span class="n">userspace</span> <span class="n">zipkin</span><span class="p">:</span><span class="n">timestamp</span>
<span class="n">lttng</span> <span class="n">enable</span><span class="o">-</span><span class="n">event</span> <span class="o">--</span><span class="n">userspace</span> <span class="n">zipkin</span><span class="p">:</span><span class="n">keyval_integer</span>
<span class="n">lttng</span> <span class="n">enable</span><span class="o">-</span><span class="n">event</span> <span class="o">--</span><span class="n">userspace</span> <span class="n">zipkin</span><span class="p">:</span><span class="n">keyval_string</span>
<span class="n">lttng</span> <span class="n">start</span>
</pre></div>
</div>
<p>Then start up Ceph again.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">OSD</span><span class="o">=</span><span class="mi">3</span> <span class="n">MON</span><span class="o">=</span><span class="mi">3</span> <span class="n">RGW</span><span class="o">=</span><span class="mi">1</span> <span class="o">../</span><span class="n">src</span><span class="o">/</span><span class="n">vstart</span><span class="o">.</span><span class="n">sh</span> <span class="o">-</span><span class="n">n</span> <span class="o">-</span><span class="n">o</span> <span class="s2">&quot;rbd_blkin_trace_all&quot;</span>
</pre></div>
</div>
<p>You may want to check that ceph is up.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">status</span>
</pre></div>
</div>
<p>Now put something in using rados, check that it made it, get it back, and remove it.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">osd</span> <span class="n">pool</span> <span class="n">create</span> <span class="n">test</span><span class="o">-</span><span class="n">blkin</span>
<span class="n">rados</span> <span class="n">put</span> <span class="n">test</span><span class="o">-</span><span class="nb">object</span><span class="o">-</span><span class="mi">1</span> <span class="o">../</span><span class="n">src</span><span class="o">/</span><span class="n">vstart</span><span class="o">.</span><span class="n">sh</span> <span class="o">--</span><span class="n">pool</span><span class="o">=</span><span class="n">test</span><span class="o">-</span><span class="n">blkin</span>
<span class="n">rados</span> <span class="o">-</span><span class="n">p</span> <span class="n">test</span><span class="o">-</span><span class="n">blkin</span> <span class="n">ls</span>
<span class="n">ceph</span> <span class="n">osd</span> <span class="nb">map</span> <span class="n">test</span><span class="o">-</span><span class="n">blkin</span> <span class="n">test</span><span class="o">-</span><span class="nb">object</span><span class="o">-</span><span class="mi">1</span>
<span class="n">rados</span> <span class="n">get</span> <span class="n">test</span><span class="o">-</span><span class="nb">object</span><span class="o">-</span><span class="mi">1</span> <span class="o">./</span><span class="n">vstart</span><span class="o">-</span><span class="n">copy</span><span class="o">.</span><span class="n">sh</span> <span class="o">--</span><span class="n">pool</span><span class="o">=</span><span class="n">test</span><span class="o">-</span><span class="n">blkin</span>
<span class="n">md5sum</span> <span class="n">vstart</span><span class="o">*</span>
<span class="n">rados</span> <span class="n">rm</span> <span class="n">test</span><span class="o">-</span><span class="nb">object</span><span class="o">-</span><span class="mi">1</span> <span class="o">--</span><span class="n">pool</span><span class="o">=</span><span class="n">test</span><span class="o">-</span><span class="n">blkin</span>
</pre></div>
</div>
<p>You could also use the example in <code class="docutils literal notranslate"><span class="pre">examples/librados/</span></code> or <code class="docutils literal notranslate"><span class="pre">rados</span> <span class="pre">bench</span></code>.</p>
<p>Then stop the LTTng session and see what was collected.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lttng</span> <span class="n">stop</span>
<span class="n">lttng</span> <span class="n">view</span>
</pre></div>
</div>
<p>You’ll see something like::</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="mi">15</span><span class="p">:</span><span class="mi">33</span><span class="p">:</span><span class="mf">08.884275486</span><span class="p">]</span> <span class="p">(</span><span class="o">+</span><span class="mf">0.000225472</span><span class="p">)</span> <span class="n">ubuntu</span> <span class="n">zipkin</span><span class="p">:</span><span class="n">timestamp</span><span class="p">:</span> <span class="p">{</span> <span class="n">cpu_id</span> <span class="o">=</span> <span class="mi">53</span> <span class="p">},</span> <span class="p">{</span> <span class="n">trace_name</span> <span class="o">=</span> <span class="s2">&quot;op&quot;</span><span class="p">,</span> <span class="n">service_name</span> <span class="o">=</span> <span class="s2">&quot;Objecter&quot;</span><span class="p">,</span> <span class="n">port_no</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">ip</span> <span class="o">=</span> <span class="s2">&quot;0.0.0.0&quot;</span><span class="p">,</span> <span class="n">trace_id</span> <span class="o">=</span> <span class="mi">5485970765435202833</span><span class="p">,</span> <span class="n">span_id</span> <span class="o">=</span> <span class="mi">5485970765435202833</span><span class="p">,</span> <span class="n">parent_span_id</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">event</span> <span class="o">=</span> <span class="s2">&quot;osd op reply&quot;</span> <span class="p">}</span>
<span class="p">[</span><span class="mi">15</span><span class="p">:</span><span class="mi">33</span><span class="p">:</span><span class="mf">08.884614135</span><span class="p">]</span> <span class="p">(</span><span class="o">+</span><span class="mf">0.000002839</span><span class="p">)</span> <span class="n">ubuntu</span> <span class="n">zipkin</span><span class="p">:</span><span class="n">keyval_integer</span><span class="p">:</span> <span class="p">{</span> <span class="n">cpu_id</span> <span class="o">=</span> <span class="mi">10</span> <span class="p">},</span> <span class="p">{</span> <span class="n">trace_name</span> <span class="o">=</span> <span class="s2">&quot;&quot;</span><span class="p">,</span> <span class="n">service_name</span> <span class="o">=</span> <span class="s2">&quot;Messenger&quot;</span><span class="p">,</span> <span class="n">port_no</span> <span class="o">=</span> <span class="mi">6805</span><span class="p">,</span> <span class="n">ip</span> <span class="o">=</span> <span class="s2">&quot;0.0.0.0&quot;</span><span class="p">,</span> <span class="n">trace_id</span> <span class="o">=</span> <span class="mi">7381732770245808782</span><span class="p">,</span> <span class="n">span_id</span> <span class="o">=</span> <span class="mi">7387710183742669839</span><span class="p">,</span> <span class="n">parent_span_id</span> <span class="o">=</span> <span class="mi">1205040135881905799</span><span class="p">,</span> <span class="n">key</span> <span class="o">=</span> <span class="s2">&quot;tid&quot;</span><span class="p">,</span> <span class="n">val</span> <span class="o">=</span> <span class="mi">2</span> <span class="p">}</span>
<span class="p">[</span><span class="mi">15</span><span class="p">:</span><span class="mi">33</span><span class="p">:</span><span class="mf">08.884616431</span><span class="p">]</span> <span class="p">(</span><span class="o">+</span><span class="mf">0.000002296</span><span class="p">)</span> <span class="n">ubuntu</span> <span class="n">zipkin</span><span class="p">:</span><span class="n">keyval_string</span><span class="p">:</span> <span class="p">{</span> <span class="n">cpu_id</span> <span class="o">=</span> <span class="mi">10</span> <span class="p">},</span> <span class="p">{</span> <span class="n">trace_name</span> <span class="o">=</span> <span class="s2">&quot;&quot;</span><span class="p">,</span> <span class="n">service_name</span> <span class="o">=</span> <span class="s2">&quot;Messenger&quot;</span><span class="p">,</span> <span class="n">port_no</span> <span class="o">=</span> <span class="mi">6805</span><span class="p">,</span> <span class="n">ip</span> <span class="o">=</span> <span class="s2">&quot;0.0.0.0&quot;</span><span class="p">,</span> <span class="n">trace_id</span> <span class="o">=</span> <span class="mi">7381732770245808782</span><span class="p">,</span> <span class="n">span_id</span> <span class="o">=</span> <span class="mi">7387710183742669839</span><span class="p">,</span> <span class="n">parent_span_id</span> <span class="o">=</span> <span class="mi">1205040135881905799</span><span class="p">,</span> <span class="n">key</span> <span class="o">=</span> <span class="s2">&quot;entity type&quot;</span><span class="p">,</span> <span class="n">val</span> <span class="o">=</span> <span class="s2">&quot;client&quot;</span> <span class="p">}</span>
</pre></div>
</div>
</section>
<section id="install-zipkin">
<h2>Install  Zipkin<a class="headerlink" href="#install-zipkin" title="Permalink to this heading"></a></h2>
<p>One of the points of using Blkin is so that you can look at the traces
using Zipkin. Users should run Zipkin as a tracepoints collector and
also a web service. The executable jar runs a collector on port 9410 and
the web interface on port 9411</p>
<p>Download Zipkin Package:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">git</span> <span class="n">clone</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">openzipkin</span><span class="o">/</span><span class="n">zipkin</span> <span class="o">&amp;&amp;</span> <span class="n">cd</span> <span class="n">zipkin</span>
<span class="n">wget</span> <span class="o">-</span><span class="n">O</span> <span class="n">zipkin</span><span class="o">.</span><span class="n">jar</span> <span class="s1">&#39;https://search.maven.org/remote_content?g=io.zipkin.java&amp;a=zipkin-server&amp;v=LATEST&amp;c=exec&#39;</span>
<span class="n">java</span> <span class="o">-</span><span class="n">jar</span> <span class="n">zipkin</span><span class="o">.</span><span class="n">jar</span>
</pre></div>
</div>
<p>Or, launch docker image:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">docker</span> <span class="n">run</span> <span class="o">-</span><span class="n">d</span> <span class="o">-</span><span class="n">p</span> <span class="mi">9411</span><span class="p">:</span><span class="mi">9411</span> <span class="n">openzipkin</span><span class="o">/</span><span class="n">Zipkin</span>
</pre></div>
</div>
</section>
<section id="show-ceph-s-blkin-traces-in-zipkin-web">
<h2>Show Ceph’s Blkin Traces in Zipkin-web<a class="headerlink" href="#show-ceph-s-blkin-traces-in-zipkin-web" title="Permalink to this heading"></a></h2>
<p>Download babeltrace-zipkin project. This project takes the traces
generated with blkin and sends them to a Zipkin collector using scribe:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">git</span> <span class="n">clone</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">vears91</span><span class="o">/</span><span class="n">babeltrace</span><span class="o">-</span><span class="n">zipkin</span>
<span class="n">cd</span> <span class="n">babeltrace</span><span class="o">-</span><span class="n">zipkin</span>
</pre></div>
</div>
<p>Send lttng data to Zipkin:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>python3 babeltrace_zipkin.py ${lttng-traces-dir}/${blkin-test}/ust/uid/0/64-bit/ -p ${zipkin-collector-port(9410 by default)} -s ${zipkin-collector-ip}
</pre></div>
</div>
<p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">python3</span> <span class="n">babeltrace_zipkin</span><span class="o">.</span><span class="n">py</span> <span class="o">~/</span><span class="n">lttng</span><span class="o">-</span><span class="n">traces</span><span class="o">-</span><span class="nb">dir</span><span class="o">/</span><span class="n">blkin</span><span class="o">-</span><span class="n">test</span><span class="o">-</span><span class="mi">20150225</span><span class="o">-</span><span class="mi">160222</span><span class="o">/</span><span class="n">ust</span><span class="o">/</span><span class="n">uid</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mi">64</span><span class="o">-</span><span class="n">bit</span><span class="o">/</span> <span class="o">-</span><span class="n">p</span> <span class="mi">9410</span> <span class="o">-</span><span class="n">s</span> <span class="mf">127.0.0.1</span>
</pre></div>
</div>
<p>Check Ceph traces on webpage:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>Browse http://${zipkin-collector-ip}:9411
Click &quot;Find traces&quot;
</pre></div>
</div>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../balancer-design/" class="btn btn-neutral float-left" title="Ceph 如何均衡（读写、容量）" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../bluestore/" class="btn btn-neutral float-right" title="BlueStore Internals" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>